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Abstract. Distributions of surface orientation and reflectance factor on the surface of an object 
can be determined from scene radiances observed by a fixed sensor under varying lighting 
conditions. Such techniques have potential application to the automatic inspection of industrial 
parts, the determination of the attitude of a rigid body in space and the analysis of images 
returned from planetary explorers. A comparison is made of this method with techniques based on 
images obtained from different viewpoints with fixed lighting 
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INTRODUCTION 

Reflected radiance depends on incident radiance, surface orientation and the properties 
of the surface material. An imaging device produces image irradiances proportional to scene 
radiances. Consequently the gray levels, quantized measurements of the image irradiance values, 
contain a great deal of information about the scene being imaged. This information is present in 
the raw gray levels in a coded form however. To learn more about the objects being viewed, it is 
necessary to carefully analyze the gray levels in order to infer the surface properties of the objects. 
This is not easy since a single gray level value depends on multiple factors. To constrain possible 
solutions to this problem it is helpful to have available more than one image taken from the same 
viewpoint but with varied lighting conditions. At each point in the image one then has more than 
one constraint on the possible surface orientation and surface reflectance factor. 

Such reconstructions of a scene being viewed depend on a detailed understanding of the 
imaging process. In addition, the imaging instrument must be of high caliber, so that the gray 
levels produced can be dependably related to scene radiances. Fortunately our understanding of 
image formation and the physics of light reflection has advanced sufficiently, and the quality of 
imaging devices is now high enough, to make this endeavour feasible. The details of a number of 
particular cases have been developed. Much remains to be done, particularly testing of these ideas 
on images of practical importance. Further analytic development will also be required to define the 
exact conditions for which unambiguous solutions are possible and to construct algorithms that can 
find these solutions efficiently. 

It will perhaps be useful to list here some cases that have been explored so far. 
(1) Determining the shape from a single image 

Understanding of the human visual system, and the construction of artificial vision 
systems, require an appreciation of shading, the variation of reflected light intensity with surface 
orientation. The scene radiance seen by the image sensor is the product of the scene irradiance, 
and the reflectance. The reflectance in turn is a function of the direction of the surface normal 
relative to the direction to the light source and the direction to the image sensor. The reflectance 
function discussed here can be related to the bidirectional reflectance-distribution function (BRDF) 
defined by the National Bureau of Standards [11 It depends on the material of which the surface 
is composed as well as its microstructure. I 

Let R(p, q) be the scene radiance of a surface element with gradient (p, q), where p and 
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q are the first partial derivatives of the surface elevation with respect to coordinate axes parallel to 
the image axes. We have called R(p, q) the "reflectance map" [2, Si It gives scene radiance as a 
function of surface gradient and can be determined from the BRDF for the surface material and 
the distribution of incident radiance. If the distribution of lightsources is known, and the object 
has a uniform surface covering, its shape can be calculated from the image irradiances in a single 
image [2, 4l If I(x, y) is the image irradiance at the image point (x, y), then the relevant equation 

is 

■ i 

■; . . |. ■ . ■ 

I(x, y) = R(p, q). (1) 

Until recently, the calculation of shape from shading could be performed only by a 
rather tedious method involving the direct solution of the above nonlinear first-order partial 
differential equation, using something like the method of characteristic strip expansion. Progress 
has been made in the development of an iterative, local method based on relaxation or 
"cooperative computation" [3, 51 It is interesting, by the way, that a special case applicable to the 
material in the maria of the moon had been solved earlier by Rindfleisch [61 This in turn was 
inspired by work on the special case of surfaces with very shallow angles between the local tangent 
plane and the incident light [71 The method of Rindfleisch, incidentally, was based on an 
t '*! assumed reflectance function that violates basic physical laws [81 

(2) Determining shape from two images obtained with different lighting. 

If two images taken from the same point of view are available, then two constraints 
apply at every image point. The possible directions of the local surface normal form a two 
parameter family. It seems reasonable then that enough information might be available to find 
the surface normal by local computations. This method, which has been called photometric stereo 
[5, 9, 10], produces information about the shape of an object in the form of a distribution of 
surface normals. This kind of representation for shape has been favored recently in the machine 
vision community because of several advantageous properties [2, 111 As an object rotates, for 
example, the surface normals undergo a rather simple transformation, while distances to the 
surface change in less symmetrical fashion. Here one obtains a pair of equations, 

Il(x, y) = Rj(p, q) (2) 

I 2 (x, y) * R 2 (p, q) (3) 

The calculation of surface normal from the two image irradiances at each point is local 

/~\ and can be implemented by means of a simple two dimensional look-up table. This table in 

essence is the inverse of a table containing the two image irradiances indexed on the two 



PAGE 4 



r^. 



/^\ 



components of the surface normal. The calculation is local, rapid and not subject to possible error 
propagation as is the serial method for single images mentioned above. There may be some 
ambiguity that cannot be resolved locally, since more than one surface normal orientation may 
produce the same pair of scene radiances. This difficulty may be resolved by an iterative 
"relaxation* computation in which compatible solutions for neighbouring points are reinforced 
while conflicting assignments are eliminated. 

It is possible to produce inconsistent surface normals as a result of noise in the scene 
radiance measurements or as a result of depth discontinuities (perhaps even due to discontinuities 
in gradient). This possibility arises because the surface normal is perpendicular to a local tangent 
plane, which in turn is defined by the partial derivatives of distance to the object's surface with 
respect to two orthogonal directions parallel to the coordinate axes of the image plane. From the 
representation in terms of surface normals one can thus calculate the gradient t2] of the surface. 
For a smooth surface the partial derivatives along orthogonal directions are not independent - the 
second partial derivative with respect to the two directions taken in one order must equal the 
second partial derivative obtained with the reverse order. 

Little has been done to explore methods that use violations of this condition to segment 
the image into regions belonging to different objects, or to iteratively refine the solution using the 
redundant information available in areas where there are no depth discontinuities. 

(3) Determining shape and reflectance using two images. 

In many real situations the objects do not have uniform surface properties. In this case 
the methods described above will be misled, much as a human is by carefully applied facial 
make-up. It is thus of interest to explore methods that can solve for surface reflectance factor and 
shape simultaneously. There is not enough information in a single image to accomplish this task 
in general. Under appropriate circumstances it is not possible to distinguish a photographic print 
from the scene portrayed in the photograph. The same image irradiances are produced in a 
monocular image sensor in the one case by a flat surface with varying reflectance and in the other 
case by three-dimensional shapes that may have uniform surface reflectance properties. 

Two images do provide enough information however. Indeed, it was apparent in the 
previous section that two images contain more information than is required to solve for shape 
alone. If the surface reflectance factor or •albedo* is p(x, y) then one obtains two equations, 

*o h^ Xt y) s p< x » y) * r i<p» q) (4> 

i 2 (x, y) * P(x, y) * R 2 (p, q) (5) 
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The variable surface reflectance factor can be removed by dividing the two images 
(provided the reflectance factor or "albedo" appears as a multiplicative factor). If we let I J2 be the 
ratio of 1 { and I 2 , and R 12 be the ratio of Rj and R 2 , then 

I 12 (x, y) - R 12 (p, q) (6) 

To this peculiar, new "image" one can then apply the shape from shading methods 
mentioned in section (1) above for the determination of shape from a single image. Finally, one 
can recover the reflectance factor by dividing one of the real images by a synthetic image obtained 
using the shape so determined [121 

p(x, y) = i,(x, y)/S,(x, y) (7) 

Here Sj is a synthetic image computed using the reflectance map Rj(p, q). In practice/ to 
assure numerical accuracy, it would be better to perform this operation on the sum of the two 
images (using the sum of the two reflectance maps). The spatial distribution of surface gradient 
obtained in this fashion, as well as the spatial distribution of reflectance factor, are examples of 
"intrinsic images" [13], distributions of underlying information that can be extracted from the raw 
image irradiances. The method described here can be criticized on the same grounds as the basic 
shape from shading method in that it requires numerical solution of a non-linear first-order partial 
differential equation. Also, more complicated methods are required if the varying reflectance is 
caused by variations in surface cover that cannot be modelled simply as multiplication by a 
varying reflectance factor. 

(4) Determining shape and reflectance using three images. 

When three images taken under different lighting conditions are available, enough 
constraint is available to solve for the surface normal and reflectance factor locally. In fact, three 
pieces of information are available at every point and three variables are needed to pin down the 
two components of local surface orientation and the reflectance factor. If the reflectance function 
has a particularly simple form an analytic solution for the surface normal and the reflectance 
factor is possible [10]. If this is not the case, one may use a simple algorithm based on a 
three-dimensional look-up table indexed by the three gray levels found in the images at 
corresponding image points. Here the three equations of interest are, 

Ii(x, y) = p(x, y) * Rj(p, q) (8) 

I 2 (x, y) = p(x, y) * R 2 (p, q) (9) 

I 3 (x, y) = p(x, y) * R 3 (p, q) (10) 



PAGE 6 



r\ 



A better method again uses ratios of images to remove the variable surface reflectance 
factor. If for example two new "images" are generated by dividing two pairs of the original three 
images, these can be used to determine the shape as described above in section (2). If I| 2 is the 
ratio of Ij and I 2 and so on, then the relevant equations are, 

I 12 (x, y) * R 12 (p f q) (H) 

I 23 (x, y) * R 23 (p, q) (12) 

A two dimensional look-up table is appropriate for this computation. The surface 
reflectance factor can once again be determined after the surface shape has been found, by 
division of one of the real images with a synthetic image created using the shape information. For 
reasons of numerical accuracy it is actually more appropriate to use other combinations of two 
images than their ratio - it may be convenient to work with the ratio of their difference and their 
sum for example. 
I 

! 

[I 2 - l,]/[l 2 ♦ 111 =[R 2 - R iHR 2 + R il <»> 

Many details remain to be worked out. It appears for example that the positions of the 
r^ light sources must be choosen carefully so as to insure that the three measurements of image 

intensity are in fact independent [101 When the directions from the objects being viewed to the 
light sources all lie in a plane no new information may be obtained from the third image. 



Comparison with ordinary stereo methods 

Shading is not usually the first depth cue that comes to mind. Stereo, based on two 
images taken from different points of view, with the same lighting, will usually be mentioned first. 
This is curious, since shading in many cases is the more important cue, particularly for printed 
material containing pictures of smooth, rounded objects such as people. Nevertheless, stereo is a 
method of great practical importance, since topographic maps and digital terrain models D4] are 
derived from aerial photography by this technique. 

Elevations can be found by a human operator who fuses corresponding image details 
from the two pictures [15, 16], or by means of "automated" methods, usually based on correlation [17, 
18, 19, 201 While automated methods still leave miich to be desired in terms of robustness in the 
face of highly variable terrain and terrain cover (or smooth areas without surface markings), many 
^^ ideas from the machine vision community point to more sophisticated techniques for the solution 

of the correspondence problem [21, 22, 23, 241 At the cost of greater computational effort, these 
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promise to produce excellent results. While much remains to be done, it seems likely that really 
automated stereo machines will eventually be commonplace. 

Methods using radiometric measurements in one or more image from the same viewpoint 
to determine surface shape and reflectance factor can be considered complementary to methods 
based on the identification of corresponding points in two images taken from different viewpoints: 

(1) Stereo allows the accurate determinations of distances to objects. Radiometric 
methods on the other hand are best when the surface gradient is to be found. 

(2) Stereo works well on rough surfaces with discontinuities of surface orientation. 
Radiometric methods work best on smooth surfaces with few discontinuities. 

(3) Stereo does well with "painted" or textured surfaces with varying surface 
reflectance. Radiometric methods are best when applied to surfaces with 
uniform surface properties. 

Methods based on analysis of radiometric measurements do have some unique advantages 
f^\ however: 

(1) There is no difficulty identifying points in the two images that correspond to 
the same object point since the images are taken from the same point of view. 
This is the major computational task in the analysis of stereo information. 

(2) Under appropriate circumstances, the surface reflectance factor can be 
determined because the effect of surface orientation on scene radiance can be 
removed. Stereo provides no such capability. 

(3) Describing the shape of an object in terms of its surface normal is preferable in 
a number of situations to description in terms of elevation about some reference 
plane. This is the natural form of output from the radiometric methods. 
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